Marking essays on screen: An investigation into the reliability of marking extended subjective texts

There is a growing body of research literature that considers how the mode of assessment, either computeror paper-based, might affect candidates’ performances (Paek, 2005). Despite this, there is a fairly narrow literature that shifts the focus of attention to those making assessment judgments and which considers issues of assessor consistency when dealing with extended textual answers in different modes. This research project explored whether the mode in which a set of extended essay texts were accessed and read systematically influenced the assessment judgments made about them. During the project twelve experienced English Literature assessors marked two matched samples of ninety essay exam scripts on screen and on paper. A variety of statistical methods were used to compare the reliability of the essay marks given by the assessors across modes. It was found that mode did not present a systematic influence on marking reliability. The analyses also compared examiners’ marks with a gold standard mark for each essay and found no shifts in the location of the standard of recognised attainment across modes. Introduction Literature suggests that the feasibility, validity and reliability of working on-screen have long been the focus of audiences from a wide range of backgrounds, ranging from studies in the contexts of education, to those in occupational and cognitive psychology, among others. This particular study sought to investigate whether the mode of marking (on-screen or paper) had any influence on essay marking reliability and markers’ leniency/rigour. Literature Review Bennett (2002) describes the rapid growth of computer technology use in workplaces and education as inexorable. Although technology offers the potential to broaden educational assessment beyond what traditional methods allow, there are inevitable concerns during a transition phase (where assessments exist in both paperand computer-based modes) that their outcomes are not comparable. In her review of comparability studies since 1993 Paek (2005) notes that the transition from paperto computer-based testing cannot be taken for granted and that comparability between the two testing modes needs to be established through carefully designed empirical work. She goes on to suggest that: Comparability studies explore the possibility of differential effects due to the use of computer-based tests instead of paper-and-pencils tests. These studies help ensure that test score interpretations remain valid and that students are not disadvantaged in any way by taking a computerized test instead of the typical paper test. (p.1) Gathering reliability measures is one significant practical step towards demonstrating the validity of computer-based testing during the transitional phase.